Test Spy

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 538 of xUnit Test Patterns for the latest information.
Also known as: Spy, Test Spy

Variation of: Test Double

How do we implement Behavior Verification?
How can we verify logic independently when it has indirect outputs to other software components?

Use a Test Double to capture the indirect output calls made to another component by the system under test (SUT) for later verification by the test.

Sketch Test Spy embedded from Test Spy.gif

In many circumstances, the environment or context in which the SUT operates very much influences the behavior of the SUT. To get good enough visibility of the indirect outputs of the SUT, we may have to replace some of the context with something we can use to capture these outputs of the SUT.

Use of a Test Spy is a simple and intuitive way to implement an observation point that exposes the indirect outputs of the SUT so they can be verified.

How It Works

Before we exercise the SUT, we install a Test Spy as a stand-in for depended-on component (DOC) used by the SUT. The Test Spy is designed to act as an observation point by recording the method calls made to it by the SUT as it is exercised. During the result verification phase, the test compares the actual values passed to the Test Spy by the SUT with the values expected by the test.

When To Use It

A key indication for using a Test Spy is having an Untested Requirement (see Production Bugs on page X) caused by an inability to observe side-effects of invoking methods on the SUT. Test Spys are a natural and intuitive way to extend the existing tests to also cover these indirect outputs because the calls to the Assertion Methods (page X) are invoked by the test after the SUT has been exercised just like in "normal" tests. The Test Spy merely acts as the observation point that gives the Test Method (page X) access to the values recorded during the SUT execution.

We should use a Test Spy if any of the following are true:

We are verifying the indirect outputs of the SUT and we can cannot predict the value of all attributes of the interactions with the SUT ahead of time.
We want the assertions to be visible in the test and we don't think the setting up of the Mock Object (page X) expectations is sufficiently intent-revealing.
Our test requires test-specific equality therefore we cannot use the standard definition of equality as implemented in the SUT and we are using tools that generate the Mock Object but which do not give us control over the Assertion Methods that are being called.
A failed assertion cannot be reported effectively back to the Test Runner (page X). This might occur if the SUT is running inside a container that catches all exceptions and makes it difficult to report the results or if the logic of the SUT runs in a different thread or process from the test which invokes it. (Both of these cases really beg refactoring to allow us to test the SUT logic directly, but that is the subject of another chapter.)

If none of these apply, we may want to consider using a Mock Object. If we are trying to address Untested Code (see Production Bugs) by controlling the indirect inputs of the SUT, a simple Test Stub (page X) may be all we need.

Because a Test Spy does not fail the test at the first deviation from the expected behavior like a Mock Object, our tests will be able to include more detailed diagnostic information in the Assertion Message (page X) based on information gathered after a Mock Object would have failed the test. But at the point of test failure, only the information within the Test Method itself is available to be used in the calls to the Assertion Methods. If we need to include information that is only accessible while the SUT is being exercised, we'll have to use a Mock Object.

Of course, we won't be able to use any Test Double (page X) unless the SUT implements some form of substitutable dependency.

Implementation Notes

The Test Spy itself can be built as a Hard-Coded Test Double (page X) or as a Configurable Test Double (page X). Since I've given detailed examples in those patterns, I'll only provide a quick summary here. Likewise, we can use any of the substitutable dependency patterns to install it before we exercise the SUT.

The key characteristic in how a Test Spy is used by a test is that assertions are done from within the Test Method. Therefore, the test must recover the indirect outputs captured by the Test Spy before it can do its assertions. This can be done in several ways:

Variation: Retrieval Interface

We can define a Retrieval Interface on the Test Spy that exposes the recorded information. After the test has exercised the SUT, it use the Retrieval Interface to retrieve the actual indirect outputs of the SUT from the Test Spy and then calls Assertion Method with them as arguments.

Variation: Self Shunt

Also known as: Loopback

We can collapse the Test Spy and the Testcase Class (page X) into a single object called a Self Shunt. The Test Method installs itself, the Testcase Object (page X), as the DOC into the SUT. Whenever the SUT delegates to the DOC, it is actually calling methods on the Testcase Object which implements the methods by saving the actual values into instance variables that can be accessed by the Test Method. The methods could also do assertions in the inner class in which case it is a variation on Mock Object rather than Test Spy. In statically typed languages, the Testcase Class must also implement the outgoing interface (the observation point) on which the SUT depends.

Variation: Inner Test Double

A popular way to implement the Test Spy as a Hard-Coded Test Double is to code it as an anonymous inner class or block within the Test Method and have it save the actual values into instance or local variables accessible by the Test Method. This is really another way to implement a Self Shunt (see Hard-Coded Test Double).

Variation: Indirect Output Registry

Yet another possibility is to have the Test Spy store the actual parameters in a well-known place where the Test Method can access them. For example, the Test Spy could save them in a file or in a Registry[PEAA] object.

Motivating Example

The following test verifies the basic functionality of removing a flight but it does not verify the indirect outputs of the SUT, namely, the fact that it is expected to log each time a flight is removed along with data/time and username of the requester.

   public void testRemoveFlight() throws Exception {
      // setup
      FlightDto expectedFlightDto = createARegisteredFlight();
      FlightManagementFacade facade = new FlightManagementFacadeImpl();
      // exercise
      facade.removeFlight(expectedFlightDto.getFlightNumber());
      // verify
      assertFalse("flight should not exist after being removed",
                  facade.flightExists( expectedFlightDto.getFlightNumber()));
   }
Example UntestedRequirementTest embedded from java/com/clrstream/ex8/test/FlightManagementFacadeTest.java

Refactoring Notes

We can add verification of indirect outputs to existing tests using a Replace Dependency with Test Double (page X) refactoring. This involves adding code to the fixture setup logic of the tests to create the Test Spy, configuring it with values to return and installing it. At the tend of the test, we add assertions comparing the expected method names and arguments of the indirect outputs with the actual values retrieved from the Test Spy using the Retrieval Interface.

Example: Test Spy

In this improved version of the test, logSpy is our Test Spy. The statement facade.setAuditLog(logSpy) installs the Test Spy using the Setter Injection (see Dependency Injection on page X) pattern. The methods getDate, getActionCode, etc. are the Retrieval Interface used to retrieve the actual arguments of the call to the logger.

   public void testRemoveFlightLogging_recordingTestStub() throws Exception {
      // fixture setup
      FlightDto expectedFlightDto = createAnUnregFlight();
      FlightManagementFacade facade = new FlightManagementFacadeImpl();
      //    Test Double setup
      AuditLogSpy logSpy = new AuditLogSpy();
      facade.setAuditLog(logSpy);
      // exercise
      facade.removeFlight(expectedFlightDto.getFlightNumber());
      // verify
      assertFalse("flight still exists after being removed",
                  facade.flightExists( expectedFlightDto.getFlightNumber()));
      assertEquals("number of calls", 1, logSpy.getNumberOfCalls());
      assertEquals("action code", Helper.REMOVE_FLIGHT_ACTION_CODE,
                   logSpy.getActionCode());
      assertEquals("date", helper.getTodaysDateWithoutTime(), logSpy.getDate());
      assertEquals("user", Helper.TEST_USER_NAME, logSpy.getUser());
      assertEquals("detail", expectedFlightDto.getFlightNumber(),
                   logSpy.getDetail());
   }
Example RecordingTestStubUsage embedded from java/com/clrstream/ex8/test/FlightManagementFacadeTestSolution.java

This test depends on the following definition of the Test Spy:

public class AuditLogSpy implements AuditLog {
   // Fields into which we record actual usage info
   private Date date;
   private String user;
   private String actionCode;
   private Object detail;
   private int numberOfCalls = 0;
  
   // Recording implementation of real AuditLog interface:
   public void logMessage(Date date, String user,
                          String actionCode,
                          Object detail) {
      this.date = date;
      this.user = user;
      this.actionCode = actionCode;
      this.detail = detail;
      numberOfCalls++;
   }
  
   // Retrieval Interface:
   public int getNumberOfCalls() {
      return numberOfCalls;
   }
   public Date getDate() {
      return date;
   }
   public String getUser() {
      return user;
   }
   public String getActionCode() {
      return actionCode;
   }
   public Object getDetail() {
      return detail;
   }
}
Example PassiveMockObjectDefn embedded from java/com/clrstream/ex8/test/AuditLogSpy.java

Page generated at Wed Feb 09 16:39:43 +1100 2011